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13  ABSTRACT 

If 

This  working  paper  is  designed  to  stimulate  research  into  the 
concept  of  information  system  effectiveness.  The  varied  characteris¬ 
tics  and  purposes  ascribed  to  such  systems  are  identified,  including 
the  requirement  to  support,  at  least  partially,  the  decision  process 
at  one  or  more  levels  of  activity.  Discussion  covers  the  value  of 
information  and  its  impact  un  organizational  objectives.  A  proposed 
information-decision  model  stresses  the  recurring,  dynamic  demands 
for  system  output.  Measures  of  effectiveness  are  derived  from  func¬ 
tional  objectives  in  both  qualitative  and  quantitative  terms.  Econom¬ 
ic  and  behavioral  influences  on  these  performance  indicators  are  con¬ 
sidered.  Suggested  areas  for  additional  research  include  attributes 
of  information  quality,  effects  on  decision  making,  and  relationships 
of  existing  models,  theories  or  techniques  to  information  processing.  , 
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I.  Introduction 


This  working  paper  is  intended  to  establish  a  framework  to 
encourage  research  related  to  the  effectiveness  of  information  sys¬ 
tems.  Management  and  system  users  have  become  increasingly  wary 
of  promises  about  intangible  benefits  and  underestimates  of  time 
and  resources.  Economic  conditions  have  engendered  profit  or  cost 
centers  for  service,  capital  authorization  criteria  for  development, 
and  close  scrutiny  of  operating  expenses.  Evaluation  of  performance 
is  thwarted  by  lack  of  definition  of  system  objectives,  of  standard¬ 
ization  for  development  and  processing,  and  of  formal  assessment  meth¬ 
ods.  Needs  exist  to  integrate  various  concepts  and  techniques  as 
well  as  to  intensify  study  in  several  of  these  areas.  Our  objective 
is  not  to  present  solutions  nor  to  rehash  the  work  of  others,  rather 
it  is  to  point  the  reader  and  stimulate  his  interest  toward  prior 
and  potential  future  contributions  to  knowledge  about  determining 
information  system  effectiveness.  The  ideas  presented  here  are  based 
on  a  composite  of  experiences  over  a  number  of  years  and  must  certainly 
include  assimilation  of  views  and  findings  from  many  sources  which. the 
author  has  found  to  be  meaningful  and  valid.  Insofar  as  practicable 
references  have  been  made  to  those  sources  to  give  credit  and  to  aid 
the  reader’s  further  study.  The  more  immediate  influence  on  content 
of  the  paper  has  been  a  discussion  group  among  Lehigh  University  fac¬ 
ulty  and  graduate  students  from  a  variety  of  disciplines  which  met 
regularly  during  the  1971-2  academic  yea-r.  If  this  paper  fails  to 
engender  further  dialogue  and/or  controversy,  it  will  have  failed  in 
its  purpose. 

Various  terms  and  concepts  related  to  information  systems  are 
described  in  Chapter  2.  Discussion  covers  the  major  issues  which 
limit  understanding  of  the  design,  development  and  operation  of 
computer-based  systems.  However,  the  presence  of  the  human  component 
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in  such  systems  end  its  considerable  influence  on  the  realization 
of  success  or  failure  are  recognized =  We  are  assuming  that  an  in¬ 
formation  system  may  serve  widely  differing  customer,  or  user,  re¬ 
quirements.  In  every  case,  however,  at  least  part  of  its  mission 
is  to  support  decision  making  at  one  or  more  levels  of  functional 
activity.  The  evaluation  problems  are  not  severe  for  systems  that 
stress  execution  of  administrative  and  clerical  activity  or  control 
over  processes  because  the  objectives  and  variables  are  normally  de¬ 
fined  and  constrained  in  advance.  As  the  decisions  become  non- 
prcgrammed  and  even  problem  recognition  becomes  an  issue,  the  pos¬ 
sible  alternative  courses  of  action  explode  in  quantity  and  the  ef¬ 
fect  of  an  information  system  is  concerned  with  changes  in  user  or 
decision-maker  behavior.  Measurement  of  these  effects  becomes  ex¬ 
tremely  difficult  because  objectives  are  complex,  multi -attributed 
and  normally  expressed  in  general  terms  subject  to  varied  interpre¬ 
tation  (Shepard).  In  such  cases,  the  sources  of  potentially  rele¬ 
vant  data  are  numerous  and  include  both  recorded  facts  and  judgments. 
Obtaining  and  handling  the  volume  and  variegated  forms  of  data  become 
onerous  and  expensive  tasks.  Aggregation  of  the  different  attributes 
into  an  objective  function  or  figure  of  merit  is  challenging.  Even 
more  important,  it  is  difficult  to  isolate  the  quality  of  information 
delivered  by  a  system  from  the  results  of  decisions  that  are  rendered. 
For  an  automobile,  we  can  determine  the  quality  of  the  manufactured 
product,  the  skill  of  the  driver,  and  the  existing  environmental  con¬ 
ditions  in  order  to  conjecture  about  success  or  failure  in  meeting 
customer  or  public  objectives  which  vary  in  character  and  importance. 
Similar  analysis  of  an  information-decision  system  would  indicate: 

Information  -  Quality  depends  upon  accuracy,  completeness, 

and  timeliness  of  input  data  and  the  effective¬ 
ness  of  the  processing  system. 

Decision  -  Quality  depends  upon  availability  and  content  of 
information  and  the  knowledge,  or  experience, 
and  logical  ability  of  the  decision-maker. 


Outcome  -  Results  depend  upon  the  appropriateness  and  timing 
of  the  decision(s)  relative  to  organizational 
objectives  and  the  validity  of  assumptions 
about  environmental  conditions. 

Variables  not  controlled  by  the  decision-maker  and  not  anticipated 
by  the  system  designer  often  have  more  influence  on  the  outcome  than 
performance  during  information  processing.  But  the  system  is  fre¬ 
quently  the  scapegoat  for  poor  forecasts  or  changed  requirements. 

Indeed,  inadequate  knowledge  of  the  decision-making  process  in¬ 
hibits  meaningful  design  and  evaluation  of  information  systems 
( Stuff lebeam) .  Variations  of  the  decision  process  are  discernible 
among  individuals  and  for  a  single  individual  in  different  time  frames 
and  circumstances.  A  composite,  prescriptive  approach  is  not  practi¬ 
cable  in  most  operational  situations.  Only  the  simplest,  programmed 
decision  rules  avoid  heavy  emphasis  on  subjective  estimates  of: 
(Savage,  Raiffa,  Luce  and  Raiffa,  Kriebel) 

-  States  of  nature 

-  Alternative  courses  of  action  available 

-  Payoffs  of  each  alternative 

-  Value  of  payoffs  in  terms  of  objectives 

-  Timing  appropriate  to  payoffs 

Neither  complete  ignorance  nor  perfect  information  are  realistic 
states,  rather  a  condition  of  partial  ignorance  or  uncertainty  nor¬ 
mally  exists  in  which  some  objective  facts  are  known  and  judgment 
must  fill  voids  and  aid  in  interpretation  of  facts.  This  uncer¬ 
tainty  spawns  from  the  need  to  make  personal  estimates  based  on 
judgment  and  subjective  impressions  or  from  a  lack  of  confidence  in 
available  objective  facts  (Archer,  Schlaifer).  When  is  the  ob¬ 
jective  data  a  representative  sample  of  actual  conditions?  How 
does  one  combine  these  objective  and  subjective  inputs?  When 
does  accumulated  objective  information  outweigh  subjective  esti¬ 
mates?  When  does  the  cost  of  acquiring  additional  information 
exceed  the  expected  marginal  gain  in  outcome  of  a  decision? 


In  most  cases,  these  questions  are  resolved  by  a  feeling  of  confidence 
in  the  system  output  by  its  users,  being  attenuated  by  time  pressures. 
It  has  been  suggested  that  only  clairvoyance  can  eliminate  uncertain¬ 
ty  about  these  matters  (Howard). 

Chapter  3  compiles  tome  notions  about  the  decision  system  which 
emphasize  the  dynamic,  continuous  nature  of  the  decision  process 
(Hodge  and  Hodgson).  Too  frequencly,  we  assume  that  a  decision  is  made 
and  the  problem  is  solved.  Yet  information  systems  are  justified  and 
designed  to  serve  recurring  needs  of  users  involved  in  similar  classes 
of  problems  or  in  repetitive  cycles  of: 

-  Recognizing  a  need  to  act 

-  Identifying  feasible  alternatives 

-  Selecting  among  alternatives  in  a  given  time  frame. 

In  general,  our  model  assumes  that  the  system  user  is  confronted  with 
a  series  of  decisions  for  a  problem  or  related  ones  rather  than  an  ad 
hoc  state  of  affairs.  The  need  for  information  is  then  spawned  in 
order  to  reduce  uncertainty  about  the  several  issues  discussed  in  the 
previous  paragraph  (Bedford). 

Measures  of  effectiveness  (MOE)  are  discussed  in  Chapter  4  in  an 
attempt  to  link  functional  objectives  more  clearly  with  the  parameters 
that  are  identifiable  in  information-decision  system  output.  In  so 
doing,  scenarios  of  functional  activity  and  system  utilization  may 
prove  invaluable  for  identifying  the  key  performance  factors  which  in¬ 
fluence  or  signal  success  in  achieving  the  objectives  which  an  informa¬ 
tion  system  is  intended  to  support.  Emphasis  is  placed  on  tangible 
measures  that  can  be  identified  and  the  aggregation  of  both  qualitative 
and  quantitative  factors  in  a  utility  based  format.  Some  of  the  be¬ 
havioral  and  economic  influences  on  choice  of  measures  of  effectiveness 
are  discussed.  Special  concern  must  be  given  to  unintended  benefits, 
which  may  become  the  principal  justification  for  an  existing  system, 
and  either  advantageous  or  deleterious  side  effects  of  system  use  on 
other  functions  or  systems. 

In  chapter  5,  a  number  of  techniques  are  called  to  attention  which 
merit  further  attention  either  to  aid  research  or  to  provide  greater 
insight  into  the  information-decision  process.  This  listing  is  not 
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intended  to  be  exhaustive  nor  are  the  items  to  be  treated  as  possibly 
independent  solutions.  Measurement  of  the  effectiveness  of  an  in¬ 
formation  system  is,  in  itself,  a  complex  multi -attributed  problem. 
The  areas  of  uncertainty  discussed  earlier  are  quite  evident  and  it 
normally  appears  easier  to  avoid  evaluation  or  to  make  one  based  on 
ill-informed  judgments  rather  than  to  perform  a  thorough  assessment. 
Further  investigation  of  these  technique?  and  their  application  to 
information  systems  will  increase  insight  into  the  meaning  of  effec¬ 
tiveness  and  feasible  ways  of  measuring  it.  Perhaps  their  applica¬ 
tion  will  also  provide  greater  understanding  of  the  value  of  informa¬ 
tion  and  the  decision  process  itself. 
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II.  Nature  of  Information  Systems 


Background 

Information  processing  has  grown  rapidly  in  importance  and  fi¬ 
nancial  commitment  over  the  last  two  decades.  Unfortunately,  its 
products  are  relatively  insensitive  to  the  needs  of  the  market  place 
and  the  customers  have  shown  little  discrimination  in  determining 
whether  or  not  they  received  a  quality  product  at  a  bargain  price. 

The  product,  called  information,  is  intended  to  satisfy  requirements 
which  the  customer,  or  decision-maker,  has  articulated.  The  process 
which  generates  information  is  specified,  designed,  and  developed  in 
the  form  of  a  system,  that  is; 

A  set  of  interrelated  rules  and  procedures  for  processing  data 
into  information  in  order  to  get  or  control  action. 

Note  that  this  considers  data  to  be  raw  material  which  consti¬ 
tutes  input  to  the  refining  and  forming  process  in  order  to  develop 
a  useful  product.  The  purpose  of  an  information  system  is  not  to 
fill  archives  or  to  produce  yellowing  stacks  of  paper,  rather  it  is 
to  support  action  on  the  part  of  a  satisfied  consumer.  To  date,  that 
potential  consumer  has  had  to  accept  what  was  available  too  often, 
having  little  opportunity  to  shop  or  co  return  unsatisfactory  prod¬ 
ucts.  Contrary  to  many  beliefs,  the  product  is  perishable,  often 
having  a  well-defined  shelf  life.  Also,  the  customer  needs  are 
seasonal  and  the  market  is  extremely  dynamic  in  respect  to  both 
changes  in  customers  and  variants  in  their  needs  or  uses  for  in¬ 
formation. 

The  concept  of  an  information  system  is  frequently  confused  by 
references  to  narrow  purpose  computer  applications,  but  the  essen¬ 
tial  attributes  include: 

-  Single  organization  orientation.  The  information  system 
is  tailored  to  serve  the  objectives  of  an  organization  and 
to  meet  the  requirements  of  persons  within  the  organization. 
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-  Integral  to  the  organization.  The  information  system  is  - 
functionally  and  technically  integrated  with  the  organiza¬ 
tional  structure  and  flow  of  communications  implied  therein. 

-  One  of  a  kind.  An  information  system  should  be  designed  and 
developed  as  a  major  construction  project  rather  than  assume 
it  is  a  prototype  for  a  number  of  similar  systems. 

-  Evolutionary  change.  The  design  should  accommodate  flex¬ 
ibility  in  use  and  modular  replacement  so  that  the  system  can 
be  modified  to  meet  dynamic  requirements  while  remaining 
operational.  Obsolescence  must  be  low  because  a  complete  re¬ 
placement  model  will  rarely  be  feasible. 

-  Software  is  critical.  The  heart  of  the  information  system  is 
composed  of  procedures  and  programs.  Hardware  supports  these, 
but  related  more  to  efficiency  rather  than  effectiveness  of 
the  system. 

-  Humans  are  major  components.  Information  processing  requires 
a  man-machine  system  in  which  the  performance  of  humans  is  a 
major  influence  on  cost  and  effectiveness.  Consideration  must 
be  given  to  human  roles  during  processing  as  well  as  interfaces 
at  data  gathering  and  information  utilization. 

Table  II-l  describes  several  tentative  classifications  of  processing 
characteristics  which  are  intended  to  aid  in  conceptualizing  informa¬ 
tion  system  purposes.  Normally,  several  of  these  characteristics  will 
be  required  or  specified  in  a  system  which  meets  organisational  pur¬ 
poses. 

Failure  to  serve  the  demands  of  the  consumer  market  for  informa¬ 
tion  usually  results  in  systems  which  are  unused  or  which  consume  great 
effort  for  little  success  (Hodge  and  Hodgson).  Technical  successes 
often  fail  to  achieve  economic  benefits  or  operational  acceptance 
(McKinsey)  and  projected  growth  of  scope  and  importance  of  information 
systems  (Diebold)  is  inhibited.  False  assumptions  aie  frequently  made 
about  the  nature  and  volume  of  data  needed  by  decisionmakers  and  about 
the  ability  of  information  to  actually  improve  performance  of  an 
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Table  II-l 

Processing  Characteristics 


Management 


Communication 


Data  Base 


Monitoring 


Reference 


Scientific 


Models  to  project  and  forecast  based  on  historical 
trends  and  estimates  of  the  future.  Normally  re¬ 
quires  a  representative  sample  data  base  rather  than 
live  operational  data.  Interactive  mode  of  opera¬ 
tion  is  highly  desirable. 

Analysis  of  the  utilization  of  resources  by  func¬ 
tional  managers.  Requires  complete  actual  data. 

Mode  of  operation  is  usually  batch -oriented  unless 
decisions  are  related  to  short  cycle  period. 

Transmission  of  messages  among  remote  sites.  Re¬ 
quires  proper  distribution  of  input  without  inter¬ 
pretation  or  alteration  of  content.  Normally  re¬ 
quires  on-line  processing  with  store  and  forward 
capability. 

Collection  of  data  and  storage  in  computer-based 
files  for  future  access  to  content  or  facts.  Re¬ 
quires  frequent,  if  not  continuous,  input  of  data 
from  a  variety  of  sources.  Normally  requires  on¬ 
line  inquiry  mode  for  effective  delivery  of  output. 

Gathering  and  analysis  of  data  from  a  specific  pro¬ 
cess  or  operation  on  a  continual  basis.  Normally 
involves  on-line  capturing  of  digital  and/or  analog 
data  and  immediate  feedback  to  control  the  operation. 

Retrieval  of  documents  or  references  thereto,  often 
based  on  material  stored  in  microform,  technical 
papers,  and  other  media  not  computer-compatible. 
Initial  reference  is  often  accomplished  by  on-line 
computer  operation.  Delivery  of  the  desired  input 
is  normally  an  off-line,  batch  oriented  library 
process. 

Algorithm  oriented  processing  related  to  calculation 
for  analysis  or  estimation  of  operational  data.  In¬ 
cludes  statistical  and  engineering  calculation. 
Processing  is  normally  done  in  batch  mode  but  input 
may  be  collected  on-line  or  aggregated  for  subse¬ 
quent  processing. 
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organization  (Achoff).  Evaluation  is  correctly  identified  as  the 
weakest  milestone  (Kriebel)  in  developing  management  information  sys¬ 
tems  today.  There  is  little  wonder  that  this  is  true  when  the  cus¬ 
tomer  has  so  little  understanding  of  and  selection  in  the  product  he 
receives.  Better  evaluation  is  additionally  important  because  system 
output  is  normally  consumed  internally  by  the  organization  and  its 
low  quality  or  lack  of  use  compounds  the  burdens  of  expense  and.  dys¬ 
functional  actions. 

Value  of  Information 

Probably  more  than  enough  has  already  been  said  here  and  by  many 
others  about  the  distinction  between  data  and  information.  Unfortu¬ 
nately,  it  is  often  treated  as  a  matter  of  definition  rather  than  a 
design  philosophy  or  an  evaluation  criterion.  Effectiveness  of  an  in¬ 
formation  system  must  be  related  to  the  quality  of  its  output.  Value 
of  information  is  then  determined  by  analysis  of  both  the  effectiveness 
of  the  system  in  delivering  a  quality  product  and  the  cost  of  deliver¬ 
ing  that  output  to  the  customer. 

Quality  of  information  is  described  in  terms  of  benefits  in  a 
given  situation,  that  is  impact  on  the  behavior  of  users  or  on  the  re¬ 
sultant  action  in  a  decision  and/or  control  process.  The  benefits  are 
expressed  in  terms  of  ability  to  reduce  uncertainty  about: 

-  Need  for  action 

-  Existent  or  anticipated  states  of  nature 

-  Recognition  or  selection  of  alternative  courses  of  action 

-  Expected  pay  offs  for  each  alternative 

-  Timing  of  decision  and  action. 

Assessment  of  these  benefits  is  normally  based  on  personal  judgment, 
often  without  understanding  the  full  scope  and  capability  of  the  in¬ 
formation  system.  The  effects  of  actions  can  be  more  objectively 
determined,  but  they  are  frequently  observed  and  influenced  by  other 
activities  beyond  the  control  of  the  decision  maker.  Often,  more  than 
one  information  source  is  used  bv  a  decision-maker  and  it' is  difficult 
to  determine  the  relative  contribution  of  each. 


II 


Costs  are  rarely  as  obvious  as  charges  for  computer  services 
might  indicate.  The  user  of  the  system  may  consider  only  his  per¬ 
sonal  time  and  effort  to  access  and  utilize  information.  A  given  de¬ 
cision  situation  may  imply  employment  of  more  than  one  information  sys¬ 
tem  or  only  a  part  of  a  more  comprehensive  one.  Correct  allocation  of 
the  data  gathering,  processing  and  information  dissemination  costs 
must  be  combined  with  those  of  personal  commitment  by  the  user  to  be 
consistent  with  derived  benefits.  Over-sophistication  of  the  system 
or  unnecessary  emphasis  on  one  attribute  of  information  can  also  pro¬ 
duce  excessive  cost  relative  to  the  resultant  change  in  benefits  or 
effects  of  decisions. 

The  attributes  which  are  most  commonly  ascribed  to  information 
are  difficult  to  assess  and  almost  impossible  to  measure  separately 
(Langefors).  Rather  than  being  inherent  properties  in  system  output, 
they  are  important  only  in  the  context  of  operational  functions  served. 
Perhaps  we  should  reflect  on  the  characteristics  of  these  attributes: 

Relevance  -  Judgment  on  the  appropriateness  of  the  message  con¬ 
tent  received. 

Accuracy  -  limits  of  correctness  of  content  based  on  standards 
which  are  often  vague  and  subjective.  Sometimes  confused 
with  level  of  detail. 

Timeliness  -  Can  be  measured  in  terms  of  time  units  or  relative 
to  operational  cycle.  Adverse  situation  is  clear  when  mes¬ 
sage  is  late  or  causes  delay. 

Sufficiency  -  Judgment  about  the  scope  of  message  content  and  its 
ability  to  satisfy  all  requirements  based  on  the  user- 
estimate  of  need. 

Conciseness  -  Judgment  about  the  proportion  of  message  content 
which  is  utilized  in  a  given  situation.  Depends  upon  the 
mode  of  presentation.  May  be  represented  by  a  signal/ 
noise  ratio. 

Reliability  -  Judgment  about  the  consistency  of  the  data  source 
and  personal  confidence  of  the  user. 
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Discovery  -  Judgment  about  the  news  value  of  message  content 
subject  to  variation  in  personal  experience  and  recall. 

It  is  not  difficult  to  agree  that  an  adequate  message  has  news  value, 
bears  upon  the  problem  at  hand,  was  received  in  time,  and  satisfied 
the  user.  Determination  of  the  perspective  of  each  attribute  and  the 
value  of  information  is,  however,  tenuous  at  best.  For  example,  does 
timeliness  refer  to  preparation  of  a  report,  its  receipt  at  some 
destination,  ascertainment  that  it  is  correct,  or  its  use  by  a  decision¬ 
maker?  How  does  decay  and  shelf-life  relate  to  this  attribute? 

Let  us  assume  that  we  do  have  a  response  to  user  demand  that  is 
timed  and  structured  to  be  useful  in  a  decision  making  or  control  task. 
The  message  can  be  responsive  to  a  periodic  need,  to  a  predetermined 
exception  status,  or  to  an  ad  hoc  inquiry.  Perfect  information  would 
indicate  uncertainty  had  been  removed  from  the  situation,  that  is  the 
message(s)  received  met  all  attribute  criteria  in  relation  to  user 
background  and  experience.  It  is  more  likely,  however,  that  an  im¬ 
perfect  information  situation  will  exist.  This  can  occur  in  one  or 
more  of  several  ways,  given  messages  which  are: 

Incomplete  -  insufficient  content. 

Uncertain  -  accuracy,  reliability,  or  sufficiency  are  in  doubt. 

Incongruous  -  lack  of  relevance,  accuracy,  timeliness  or  suffi¬ 
ciency  indicates  incompatibility  of  content;  discovery  pro¬ 
motes  contradiction  with  experience  in  some  cases. 

Superfluous  -  excessive  content  which  does  not  discriminate 
relevant  or  discovery  items. 

Presence  of  these  faults  promotes  subjective  estimates  by  a  decision¬ 
maker  and  a  circumstance  in  which  he  is  satisficing  rather  than  op¬ 
timizing  on  the  basis  of  some  true  value  of  information. 

Organizational  Impact 

We  have  stressed  the  interdependence  of  an  information  system 
and  the  functional  objectives  of  the  organization  that  it  serves. 

Figure  II -2  outlines  the  major  levels  of  activity  encountered  in  an 
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organization  and  the  implied  nature  of  support  which  an  information 

i 

system  is  expected  to  provide.  Both  the  organizational  managers  and 
the  systems  designers  must  face  these  basic  concepts  in  order  to  es¬ 
tablish  objectives  for  the  system.  Several  cardinal  points  are 
usually  ignored  or  forgotten  in  the  tempest  of  development  effort, 
namely : 

-  Input  data  should  be  generated  as  a  byproduct  of  the  opera¬ 
tional  tasks  performed. 

-  Amount  of  output  and  flow  of  data  should  be  minimal  for 
purposes  served. 

-  Health  of  the  organization  depends  on  appropriate  actions, 
not  extensive  records. 

-  Planning  and  non-programmed  decisions  depend  upon  representa¬ 
tive  samples  of  activity,  not  live  reports  of  current  condi¬ 
tions  . 

Failure  to  establish  these  concepts  has  resulted  in  systems  that  are 
over-designed,  excessively  expensive  to  operate,  and  not  used  to  po¬ 
tential  by  their  intended  customers.  The  levels  of  activity  should 
be  served  by  a  cascade  of  information  processing  approaches  which  are 
highly  dependent  upon  processing  at  lov/er  levels  and  integral  with 
functional  activities.  New  information  services  are  then  justified 
on  an  incremental  basis  of  effort,  cost  and  benefits  from  likely  use. 

At  the  least,  the  design  of  an  information  system  is  a  master 
plan  which  is  developed  by  a  top  down  view  of  organizational  objec¬ 
tives.  Implementation  may  start  at  the  lower  activity  levels,  but 
only  for  those  tasks  which  relate  directly  to  priority  objectives  of 
the  organization.  Too  often,  the  information  system  is  conceived  and 
built  as  an  end  in  its  own  right  —  a  sterile,  ref-ndant  effort. 
Elegance  of  design  or  ease  in  operation  become  the  system  objectives 
rather  than  acceptance  and  utilization  by  the  prospective  customer. 

In  that  regard,  a  successful  information  system  must  be  a  compromise 
between  designer  and  manager  purposes.  Dialogue  to  arrive  at  a  com¬ 
promise  promotes  understanding  of  functional  goals,  assessment  of 
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organization  and  communication,,  and  definition  of  needed  information. 

As  in  a  computer  application,  perhaps  this  introspection  provides  a 
greater  contribution  to  success  than  implementation  of  a  perfunctorily 
correct  system. 

Although  earlier  attempts  to  develop  information  systems  have  not 
been  eminently  successful,  this  does  not  infer  that  their  impact  on 
organizations  has  been  minimal.  Issues  of  centralization  or  decen¬ 
tralization  both  of  the  organization  and  of  information  processing 
responsibility  have  been  given  substantial  attention  (Whisler,  Emery, 
Delehanty,  Brink).  Changes  in  organization  and  group  dynamics  have 
generally  occurred,  but  they  are  only  infrequently  attributed  directly 
to  the  advent  of  a  system  (Vergin,  Blose  and  Goetze,  House).  These 
have  included  staff  relocation,  realignment  of  communication  flow,  re¬ 
distribution  of  missions  and  reallocation  of  resources  (Davis).  Pat¬ 
terns  of  control  over  activity  have  shifted  because  of  the  availability 
of  new  information  sources  and  by  abdication  of  management  responsi¬ 
bility  to  system  designers.  Routine  or  clerical  tasks  have  been  as¬ 
sumed  by  systems,  middle  manager  responsibilities  have  been  squeezed, 
time  for  action  has  been  compressed,  and  more  complex  tasks  have  been 
undertaken  (Whisler,  Emery).  Methods  of  gathering  information  and 
making  decisions  have  changed  for  individuals  (Schroder,  Carroll). 

The  impact  has  also  been  evidenced  in  resistance  to  new  concepts  en¬ 
compassed  by  information  systems  *nd  in  reaction  of  management,  worker 
and  system  personnel  to  implementation  (Dickson  and  Simmons;  Dickson, 
Simmons  and  Anderson;  Brady).  Perhaps  the  most  significant  evidence 
of  the  interplay  between  information  systems  and  organization  lies, 
however,  in  the  existence  of  informal  organization  and  flows  of  com¬ 
munication.  Failure  of  the  formal  organization  to  provide  action  and 
information  as  needed  has  forced  individuals  to  seek  their  own  sources. 
To  be  successful,  a  well-designed  and  effectively  operating  information 
system  must  meet  this  challenge  and  minimize  the  informal  flow  of  in¬ 
formation,  particularly  in  areas  which  are  not  related  to  organizational 
politics. 
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Hierarchical  Views 

One  important  aspect  of  an  information  system  which  confounds 
evaluation  of  its  effectiveness  is  that  it  is  perceived  differently 
by  various  persons.  This  is  summarized  in  Figure  II -1  which  shows 
the  hierarchical  relationships.  At  the  core  is  the  computer  system 
which  focusses  on  hardware  and  software  performance.  At  the  outer 
layer  is  a  society  which  imposes  constraints  on  the  organization  and 
is  the  recipient  of  its  products  or  services.  Success  at  one  level  is 
normally  determined  in  the  next  outer  layer  which  is  partially  outside 
of  the  sphere  of  control  of  the  level  being  assessed.  Effectiveness 
and  efficiency  are  dependent  upon  performance  in  the  layer  under  view 
and  in  all  of  those  within  its  boundaries.  The  ubiquitous  systems 
analyst  can  appear  at  any  level  but,  all  too  often,  he  is  oriented  to 
the  hardware-software  issues. 

Performance  Measurement.  The  inner  view  of  the  hierarchy  relates  to 
speed  of  computer  operations,  throughput  of  the  computer  system,  ca¬ 
pacity  utilization  of  hardware,  and  response  time  to  the  user  of  the 
system.  The  scope  of  the  entire  system  and  its  timeliness  are  com¬ 
pletely  dependent  upon  performance  at  this  level.  However,  these  do 
not  guarantee  user  satisfaction  or  organizational  success.  Given 
several  methods  of  achieving  desired  benefits  or  effectiveness,  how¬ 
ever,  performance  measurement  can  compare  these  alternatives  and 
select  the  most  efficient  or  economic  one.  The  basic  measurement 
tools  are: 

-  Operation  time 

•  Figures  of  merit  based  on  memory  capacity,  word  size, 
data  transfer  rate,  main  memory  cycle,  instruction 
execution  time 

•  Mix  of  instructions,  modules  or  subroutines 

-  Program  execution 

•  Actual  applications  for  new  configuration 

•  Benchmark  problems 
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Figure  II-l 
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-  Mathematical  models 

•  Weighted  performance  factors 

•  Simulation 

“  Monitors  of  activity  and  status 

•  Hardware 

•  Software 

This  subject  area  has  received  substantial  attention  and  recent  lit¬ 
erature  contains  many  good  references  on  operating  measures  and  tech¬ 
niques  to  evaluate  systems  from  this  viewpoint.  Simulation  has 
particular  merit  in  measuring  and  predicting  performance  for  dynamic 
systems  which  interact  with  the  user  interface. 

User  Interface.  Much  has  been  written  about  the  importance  of 
this  view  of  the  system  and  most  of  this  text  refers  directly  to  it. 
The  information  system  is  a  special  case  of  man-machine  systems  and 
must  consider  the  many  human  factors  issues  (Meister  and  Rabideau). 
Emphasis  is  placed  on  articulating  and  serving  user  requirements  which 
are  responsive  to  decision-making  in  order  to  meet  organizational  ob¬ 
jectives  (Smith  and  Wolf,  Heany).  As  an  extension  of  organizational 
impact,  it  is  redundant  but  important  to  stress  the  active  participa¬ 
tion  of  the  potential  user  in  conceptualizing  and  proving  information 
systems.  Both  the  effectiveness  of  a  system  and  its  utilization  de¬ 
pend  upon  the  understanding  and  confidence  generated  at  this  level. 

Environment .  Revision  of  organizational  objectives  and  evolution 
of  information  systems  depend  upon  intelligence  from  the  external  en¬ 
vironment.  The  major  environmental  influences  on  the  effectiveness 
of  information  systems  are: 

-  External  factors 

•  Technology 

•  Market  and  competition 

•  Political  and  governmental 

•  Economic  (GNP,  employment,  productivity,  prices,  wages) 

•  Demographic  and  social 

•  Public  interest  and  acceptance 
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-  Corporate  factors 

•  Resources  available 

•  Personnel  capability 

•  Management  policy  and  perceptions  of  external  factors 

Ultimately,  the  success  of  the  information  system  must  be  gauged 
against  the  ability  of  the  organization  to  adapt  to  and  find  accep¬ 
tance  in  the  external  environment.  This  places  heavy  emphasis  on  the 
role  of  information  to  support  organizational  planning. 
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We  have  discussed  that  information  systems  must,  at  least  in 
part,  support  the  decision-making  process.  At  the  clerical  level  of 
activity,  they  tend  to  displace  manual  effort  on  repetitive  manipu¬ 
lations  which  change  slowly.  At  the  operational  and  tactical  levels, 
iterative  decisions  are  rendered  on  similar  classes  of  problems  with 
intermittently  changing  parameters  and  condition  variables.  Opera¬ 
tional  control  cycles  tend  to  be  short  and  regular  with  well-defined 
independent  and  dependent  variables.  Often  the  states  or  values  of 
independent  variables  are  continually  updated  by  transactions  gen¬ 
erated  directly  from  the  functional  task  to  be  controlled.  As  the 
level  of  activity  becomes  more  planning  oriented,  the  nature  of  the 
task  to  be  accomplished  becomes  less  structured  and  it  may  be  diffi¬ 
cult  even  to  perceive  the  problem. 

Thus,  routine  activities  may  involve  only  reaction  to  transac¬ 
tions  which  are  introduced  to  the  system  from  external  sources.  How¬ 
ever,  at  the  tactical  level  a  system  may  make  information  available 
to  aid  awareness  and  definition  of  problems  and  to  choose  among  alter¬ 
native  courses  of  action.  Figure  III-l  indicates  the  major  tasks  of 
this  decision  process  and  the  manner  of  using  information  to  support 
them.  Recognition  of  the  need  for  a  decision  to  control  or  initiate 
action  may  come  from  a  number  of  sources,  including  analysis  of  avail¬ 
able  information  related  to  ongoing  activity.  Both  identification  of 
the  decision  problem  and  selection  of  a  course  of  action  depend  upon 
repetitive  interaction  with  sources  of  information.  Each  decision¬ 
maker  must  evaluate  the  status  of  objective  facts  and  personal  knowl¬ 
edge  in  order  to  determine  his  need  for  additional  information. 

Having  identified  the  appropriate  source,  he  judges  the  expected  ad¬ 
ditional  value  in  contrast  to  the  cost  and  effort  of  acquiring  more 
information.  Unless  time  constraints  or  changed  circumstances  ter¬ 
minate  the  process,  a  course  of  action  is  normally  chosen  and  imple¬ 
mented.  Because  of  the  uncertainties  about  the  true  state  of  the 
‘  several  variables,  results  rarely  involve  complete  and  lasting  satis¬ 
faction.  This  often  suggests  a  heuristic  approach  to  identify  and 
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DECISION  PROCESS  MODEL 


select  courses  of  action  which  are  compatible  with  the  set  of  orga¬ 
nizational  objectives.  Failures  in  execution  or  proper  timing  of  the 
action  also  require  a  series  of  related  decisions  to  produce  desired 
results.  Many  systems  at  the  lower  levels  of  activity  are  intended 
to  monitor  changes  in  events  outside  the  system  and  to  react  at  re¬ 
curring  intervals  to  the  aggregate  implications  of  these  events.  Ad¬ 
ditionally,  an  iterative  decision  process  may  be  suggested  by  appear¬ 
ance  of  undesirable  side  effects  which  were  not  anticipated  by  a 
proposed  solution.  To  support  these  series  of  related  decisions,  it 
is  presumed  that  there  is  a  need  for  continual  synergism  with  various 
sources  of  information.  An  information  system  is  justified  by  stable 
requirements  for  information  or  by  high  potential  benefits  from  this 
dynamic  process. 

The  recycling  aspects  of  the  demand  for  information  and  inter¬ 
actions  with  the  decision  system  are  amplified  in  Figure  III-2.  Sig¬ 
nificant  times  are  indicated  by  the  emergence  of  an  event,  situation 
or  opportunity  which  will  require  action  (Tg);  the  optimum  point  to 
implement  an  action  (TQ);  and  the  limit  beyond  which  any  action  is 
ineffective  (TQ).  This  latter  default  limit  results  from  a  failure 
to  make  a  choice  among  possible  courses  of  action  (one  of  which  may 
be  "no  action")  rather  than  to  consciously  allow  conditions  to  ride 
as  a  best  alternative.  The  figure  then  shows  the  information  process¬ 
ing  involvement  that  emanates  from  a  decision  to  seek  information 
about  one  or  more  variables  from  several  sources.  An  inquiry  may 
define  retrieval  from  a  data  base,  initiate  data  gathering,  or  re¬ 
quest  a  report  at  its  usual  time.  The  delay  of  the  response  will  de¬ 
pend  upon  its  nature  and  the  relative  effort  to  meet  outstanding  in¬ 
quiries.  The  figure  indicates  that  repetitive  requests  are  normal 
and  that  choices  are  often  made  before  all  desired  information  is  re¬ 
ceived  and  digested.  Report  periods  may  include  time  before  the 
event  occurred  and  commonly  overlap  on  successive  requests,  having 
greater  time  span  than  the  interval  of  inquiries.  This  dynamic  pro¬ 
cess  can  become  an  entrapment  in  which  the  decision-maker  focusses 
on  information  processing,  or  its  failures,  and  defaults  on  the  needed 
decision. 
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(Event)  e-i  m,  (Optimum) 


In  short,  the  effectiveness  of  an  information  system  is  highly 
dependent  upon  a  decision  process  which  is  not  always  well  understood 
nor  amply  accommodated  by  a  system  design  which  fails  to  recognize 
the  dynamic,  open  system  characteristics.  The  model  we  propose  for 
consideration  here  is  neither  formal  nor  comprehensive.  We  do  suggest 
that  it  embodies  several  concepts  that  directly  influence  the  consumer 
demand  for  information.  The  recycling  characteristic  seems  to  apply 
whether  the  decision  relates  to  determining  objectives,  structuring 
procedures,  modifying  control  by  use  of  procedures,  or  reacting  to 
events.  It  is  apparent  that  the  need  for  a  decision  does  not  depend 
nor  wait  on  che  convenience  of  the  decision-maker  to  construct  a  list 
of  alternatives  and  their  possible  consequences  (Thompson).  Thus, 
inappropriate  decisions  and  failure  to  take  action  in  necessary  time 
are  tangible  considerations.  In  such  cases,  what  is  the  culpability 
of  an  information  system  which  responded  to  inquiries  without  undue 
delay  and  provided  output  well  worth  the  cost  of  producing  it? 
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IV.  Measures  of  Effectiveness 


We  have  discussed  earlier  the  difficulty  in  directly  assessing 
the  value  of  information  from  intrinsic  attributes  of  system  output. 

It  is  necessary,  therefore,  to  identify  parameters  or  dependent 
variables  which  indicate  the  level  of  performance  in  some  essential 
aspect  of  a  system.  Commonly,  such  a  measure  of  effectiveness  (MOE) 
is  an  operational  "  ctor  which  is  expected  to  vary  in  relation  to  the 
impact  of  an  information  system.  One  or  more  MOE  is  usually  derived 
from  each  benefit  which  the  information  system  is  expected  to  de¬ 
liver.  For  tangible  benefits,  an  MOE  should  be  quantitative  in  nature 
and  based  upon  objective  data.  Substantial  reliance  on  intangible 
benefits  to  justify  information  systems  suggests  that  some  qualitative 
factors  based  upon  subjective  opinions  are  normally  appropriate.  The 
MOEs  for  a  proposed  system  should  be  developed  during  the  system  def¬ 
inition  stages  following  the  general  procedure: 

1.  Define  functional  tasks  to  be  served  by  the  information 
system(s).  This  specifies  the  planning,  control,  super¬ 
visory,  and  clerical  involvement  of  activities  and  de¬ 
cisions  . 

2.  Define  the  goals  of  each  information  system.  This  in¬ 
cludes  the  functions  it  serves,  and  its  principal  pro¬ 
cessing  characteristics. 

3.  Develop  scenarios  of  the  operational  situation.  These 
describe  the  interactions  of  the  system(s)  and  their 
users  as  tasks  are  performed  and  various  problems  en¬ 
countered  . 

4.  Determine  performance  parameters  based  upon  operational 
considerations. 

5.  Measure  the  current  level  of  performance  for  each  pa¬ 
rameter.  Standards  and  criteria  of  satisfactory  per¬ 
formance  may  be  available. 
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6.  Estimate  improvement  of  the  level  of  performance  for  each 
parameter  after  implementation  of  the  system.  This  would 
normally  involve  a  range  of  estimates  based  on  optimistic, 
likely,  and  pessimistic  premises. 

7.  Relate  each  parameter  to  the  expected  benefits(s).  This 
involves  estimating  the  relative  contribution  of  each 
parameter  to  effectiveness  in  performance  of  tasks.  On 
this  basis,  credit  for  achieving  benefits  is  allocated 
among  efforts  which  influence  results. 

The  principal  difficulties  related  to  use  of  resulting  MOEs 
are  handling  qualitative  or  subjective  parameters,  weighting  and  ag¬ 
gregating  related  factors,  and  trade  offs  among  MOEs  measured  in  dif¬ 
ferent  units  (Hormann,  Stufflebeam).  Most  of  the  decision  problems 
encountered  deal  with  multi -attributes,  some  of  which  are  based  on 
hard  facts  and  others  on  judgment.  It  is  rare  that  the  most  sig¬ 
nificant  attributes  are  measured  in  the  same  units,  such  as  the  con¬ 
venience  of  comparing  all  benefits  and  costs  in  terms  of  dollars. 

As  we  have  discussed  earlier,  users  of  the  system  are  continually 
judging  its  worth.  An  evaluation  study  nearly  always  solicits  opin¬ 
ions  about  the  effectiveness  of  the  system,  satisfaction  in  its  use, 
and  recommendation  for  revision.  The  opinions  must  be  compared  among 
groups  of  individuals  for  each  parameter  and  the  composite  view 
scaled  in  relation  to  some  arbitrary  criteria  of  desired  performance. 
Well  defined  situations  promote  standards,  or  points  of  reference 
against  which  values  can  be  compared.  (For  instance,  a  MOE  of  "indirect/ 
direct  employees"  might  have  the  standard  value  of  .35.)  They  are  rarely 
adequate,  however,  for  the  complex  aspects  of  an  information-decision 
system.  Limits  of  desired  performance  in  such  ill -defined  circumstances 
require  consensus  in  operational  terms  for  each  MOE,  whether  qualitative 
or  quantitative  in  nature.  Additionally,  various  measures  must  be 
weighted  to  show  relative  contribution  to  achievement  of  a  benefit. 

This  involves  judgment  about  and  scaling  of  the  relationships  between 
MOEs  and  benefits.  Ordinal  scales,  usually  the  best  one  can  do  in 
such  open  systems,  are  insensitive  to  trends  in  the  level  of  an  MOE 
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and  in  relative  impact  among  several  MOEs  (Coombs).  Both  the  presence 
of  subjective  factors  and  the  lack  of  common  units  of  measure  suggest 
utility  as  the  common  denominator  in  any  logical  combination  of  MOEs 
into  an  objective  function  (Miller  and  Starr;  Spetzler;  Srinivasan; 
Stevens;  Swalm). 

It  is  important  to  realize  that  the  levels  of  activity  served  by 
an  information  system  strongly  influence  the  nature  of  MOEs  estab¬ 
lished.  Table  IV-1  shows  the  dominant  pattern  in  very  general  terms. 
The  functions  served  by  the  system  and  the  form(s)  of  processing 
utilized  suggest  benefit-oriented  MOEs.  Suggested  MOEs  for  various 
organizational  functions  apply  primarily  to  the  tactical  and  opera¬ 
tional  levels  (Stokes).  Expected  system  performance  can  also  be  de¬ 
scribed  in  a  profile  of  issues  related  to  user  requirements  or  ex¬ 
pectations.  This  is  useful  for  planning  and  evaluation  purposes  at 
all  levels  (Hare;  Smith  and  Wolf).  ItvAust  bfe  remembered  that  most 
systems  involve  a  variety  of  activity  levels,  forms  of  processing,  and 
areas  of  operational  support  which  confound  and  expand  MOE  definition. 
All  too  frequently,  however,  the  differing  character  of  performance 
parameters  at  various  levels  is  not  considered  and  evaluation  is 
focussed  from  a  microscopic  view. 

A  scenario  is  particularly  useful  in  developing  MOEs  and  in 
assessing  their  relative  contributions  before  systems  implementation. 
-It  consists  of  a  description  of  the  environment,  tasks  and  roles  asso¬ 
ciated  with  an  operational  situation.  The  sequence  of  events  and 
interactions  among  users  and  systems  are  described  for  several  typical 
tasks  or  problems.  The  script  can  center  on  a  function  with  its  sup¬ 
port  system(s)  or  on  an  information  system  with  the  populations  and 
functions  it  interfaces.  Given  initial  conditions  and  logical  se¬ 
quences  of  events  for  its  several  components,  the  scenario  can  be 
used  for  gaming  or  observation  of  man-machine  interactions.  In  par¬ 
ticular,  the  following  can  be  assessed  and  extrapolated  to  actual 
conditions : 

-  Perception  of  sources  of  information 
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*Access  ease,  available  period,  dependable  source,  suitability  to  purpose,  personal  convenience 


-  Effort  and  assistance  required  to  form  an  inquiry  and/or 
interpret  a  report. 

-  Number  of  accesses  or  reports  necessary  to  satisfy  an  inquiry. 

-  Use  of  information  by  participants 

-  Interdependence  or  redundancy  among  systems  or  functions. 

-  Options  and  possible  alternatives  at  various  times. 

-  Transient  aspects  and  fluctuations  in  conditions  or  in¬ 
formation. 

-  Likely  consequences  of  courses  of  action. 

-  Relationships  of  performance  indicators  and  goals. 

-  Sensitivity  of  results  to  assumptions  and  independent  variables. 

\  \ 

-  Realism  of  MOEs  and  standards. 

The  scenario  of  an  existing  situation  may  give  insight  to  possible 
unintended  benefits  or  changes  in  requirements  which  are  frequently 
encountered  in  established  systems  accessible  by  a  variety  of  users. 

Often  these  benefits  will  outweigh  the  initially  stated  ones  and  cause 
shifts  in  expected  user  populations  and  purposes.  For  a  meaningful 
scenario,  particular  attention  should  be  given  to  the  circumstances 
under  which  people  interact  and  seek  information  support.  This  can 
assist  in  judging  the  value  of  system  output  and  in  validating  mea¬ 
sures  of  effectiveness. 

The  procedure  for  identifying  MOEs  usually  generates  a  host  of 
candidates.  Only  those  which  correlate  highly  with  benefits  and  also 
prove  to  be  uniformly  understood  and  applied  should  be  retained. 

Hopefully,  a  weeding  and  pruning  process  will  yield  a  handful  of  sig¬ 
nificant  MOEs  meeting  the  following  criteria: 

1.  As  few  as  possible  selected. 

2.  Apply  both  to  present  and  future  projections. 

3.  Lead  to  worthwhile  ends. 

4.  Lead  to  progress  and  innovation. 
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5.  Permit  ready  comparisons  (Stokes). 

6.  Ordered  and  compacted  so  as  to  be  mutually  exclusive. 

Also,  an  overriding  factor  relates  to  determination  of  the  value  or 
level  of  performance  of  an  MOE  at  any  given  time.  The  most  success¬ 
ful  ones  will  be  included  in  system  specifications  and  will  produce 
value.*:  as  a  by-product  of  normal  functional  operations.  This  affords 
frequent  measurement  and  minimizes  both  the  cost  and  opportunity  for 
bias  associated  with  separate  data  collection  solely  for  evaluation 
purposes . 

Economic  Aspects 

Some  measures  of  effectiveness  incorporate  cost  or  financial  ben¬ 
efit  premises.  Often  these  figures  are  determined  directly  from  ac¬ 
counting  records.  To  the  extent  possible,  this  provides  convenience 
in  collecting  data  and  in  accumulating  the  various  MOE  contributions 
during  review  of  performance.  In  most  cases,  however,  the  economic 
effects  will  not  be  so  readily  available.  Then  the  estimation  of  per¬ 
formance  level  is  difficult  because  cost  and  benefit  factors  cannot 
be  compared  on  a  single  scale  of  measurement  units.  Cost  effectiveness 
assumes  that  a  single  attribute  (or  a  couple  of  compatible  ones)  will 
be  utilized  as  the  base  unit  for  an  objective  function.  Further,  cor¬ 
relation  of  input  and  effects  should  be  known  and  high.  This  approach 
can  be  meaningful  to  compare  alternatives  which  are  competing  for 
funds  at  relatively  fixed  costs,  or  which  provide  relatively  the  same 
performance  at  varied  costs.  However,  this  is  generally  an  over¬ 
simplification  for  the  complex,  multi -variable  situations  served  by 
information  systems.  Benefits  for  operational  activities  can  often 
be  expressed  in  money  terms,  but  expected  utility  of  this  return  may 
vary  substantially.  Deleterious  side  effects  which  were  not  antici¬ 
pated  are  rarely  reflected  in  MOEs  as  a  cost  or  a  limitation  on  ef¬ 
fectiveness.  The  cost  factors  for  an  information  system  should  cover 
user  involvement  and  results  from  late  or  misleading  output.  Such 
effects  are  difficult  to  express  in  tangible  terms.  These  and  re¬ 
lated  issues  complicate  the  identification  and  formulation  of  sound 
measures  of  effectiveness. 


IV  -  6 


43 


Behavioral  Aspects 


Although  measures  of  effectiveness  and  their  relationships  with 
controllable  variables  may  be  known,  individual  reactions  to  a  situa¬ 
tion  usually  vary.  Different  risk  and  utility  attitudes  cause  di¬ 
versity  in  choices  among  alternatives  based  on  the  same  information. 
Fluctuations  in  confidence  or  satisfaction  of  the  decision-maker  in 
the  system  may  alter  informally  the  weights  given  to  various  factors. 
These  may  also  cause  differences  in  tradeoff  between  subjective 
judgment  and  objective  reports  supplied  by  an  information  system. 

The  nature  of  reward  or  reinforcement  provided  after  decisions  have 
been  made  alters  the  preferences,  priorities  and  risk  philosophy  ap¬ 
plied  to  future  decisions.  These  various  influences  and  individual 
or  group  differences  cause  bias  and  deviation  from  norms  expected  to 
result  from  uniform  perception  and  use  of  performance  parameters. 

The  process  of  aggregating  a  number  of  factors  to  an  objective 
function  is  another  source  of  behavioral  influence  on  significance 
of  an  MOE.  Normally,  one  assumes  a  compensatory  tradeoff  and  combi¬ 
nation  among  the  weighted  performance  factors.  However,  in  a  given 
circumstance,  a  disjunctive  strategy  may  be  adopted  in  which  one  MOE 
dominates  all  others  and  compels  a  choice.  Similarly,  a  satisficing 
approach  to  alternative  selection  may  be  adopted.  That  is,  one  seeks 
the  decision  which  meets  the  minimum  threshold  on  all  factors  and  has 
support  from  affected  parties.  The  latter,  in  particular,  tends  to 
truncate  the  information  seeking  process  and  to  limit  the  achievement 
in  terms  of  the  chosen  MOEs  in  favor  of  a  more  expedient  disposition 
of  the  situation. 


44 


IV  -  7 


I 


References 


Ackoff,  R.  L.  ''Control,”  Ch.  6,  A  Concept  of  Corporate  Planning. 

New  York:  Wiley-Interscience,  1970. 

Arinc  Research  Corp.  "Establish  Basis  for  Evaluating  Effectiveness," 
Section. 2. 7  and  "Define  Measures  of  Effectiveness,"  Section  2.8, 
Guidebook  for  Systems  Analysis  /  Cost-Effectiveness.  (DDC  No. 

AD  688  154),  Annapolis,  Md.,  March,  1969. 


Bagstad,  C.  W.  "Measuring  the  Intangible  Benefits  of  a  Computer," 
Journal  of  Advanced  Management,  (April  1971),  pp.  49-53. 

Coombs,  C.  H.;  Raiffa,  H.;  and  Thrall,  R.  M.  "Some  Views  on  Math¬ 
ematical  Models  and  Measurement  Theory,"  Ch.  2,  Decisions  Pro¬ 
cesses.  (Thrall,  R.  M.  (ed.)).  New  York:  Wiley,  1954. 

v  *  x  Couger,  J.  D.  "Benefits  of  Data  Processing,"  Journal  of  System  Man¬ 
agement,  (Nov.  1971),  pp.  34-6. 

Eckenrode,  R.  T.  "Weighing  Multiple  Objectives,"  Management  Science, 
Vol,  12,  No.  3  (1965),  pp.  180-92.  " 

Emery,  J.  C.  Cost/Benefit  Analysis  of  Information  Systems,  SMIS 
Workshop  Report  No.  1,  The  Society  for  Management  Information 
Systems,  1971. 

Fatianow,  P.  R.  "Trade-off  Approaches  to  System  Evaluation,"  Pro¬ 
ceedings  of  the  17th  Annual  Conference  of  the  AIIE,  1966,  pp. 
293-302. 


Hare,  Van  Court.  "Analysis  for  Implementation,"  Ch.  13,  Systems 
Analysis:  A  Diagnostic  Approach .  New  York:  Harcourt,  Bruce  and 
Ward,  1967. 

Hormann,  A.  M.  Planning  by  Man -Machine  Synergism:  A  Characteriza¬ 
tion  of  Processes.  System  Development  Corp.  (DDC  No.  AD  704  810), 
March  1970. 

Kazanowski,  A.  D.  "A  Standardized  Approach  to  Cost-Effectiveness 
Evaluations,"  Ch.  7;  "Cost-Effectiveness  Fallacies  and  Miscon¬ 
ceptions  Revisited,"  Ch.  8,  Cost  Effectiveness  (English,  J.  M. 
(ed.)),  New  York:  Wiley,  1968,  pp.  113-165. 

Krauss,  L.  I.  "Project  Analysis  and  Selection,"  Ch.  2;  "Designing 
Computer-Based  MIS,"  Ch.  4,  Computer-Based  Management  Information 
Systems,  American  Management  Association,  1970. 

Lifson,  M.  W.  "Value  Theory,"  Ch.  6,  Cost  Effectiveness.  (English, 

J.  M.  (ed,)).  New  York:  Wiley,  1968,  pp.  79-112. 


45 

IV  -  8 


McDonough,  A.  M.  "Keys  to  a  Management  Information  System,”  Journal 

of  Industrial  Engineering,  Vol /  19,  No.  3  (March  1968),  pp.  viii-xii. 


Miller,  D.  W.  and  Starr,  M.  K.  /'The  Objectives  of  Decisions,”  Ch. 

3;  "The  Structure  of  Decisions,”  Ch.  4;  "When  a  Problem  is 
Worth  Solving?”  Ch.  7,  Thff Structure  of  Human  Decisions.  Engle¬ 
wood  Cliffs,  N.  J.:  Prentice -Ha 11,  1967. 

Quade,  E.  S.  "Methods  and  Procedures,”  Ch.  8,  Analysis  for  Military 
Decisions .  New  York:  North  Holland  and  Rand  McNally,  1964. 

Smith,  W.  A.,  Jr.  and  Wolf,  A.  M.,  "Generalized  Evaluation  Procedure," 
Ch.  Ill,  "Computer  System  Factors,"  "Appendix  C",  Procedures  for 
Analysis  of  Information  System  Effectiveness,  Industrial  En¬ 
gineering  Dept,  report  IE-14-7201,  sponsored  by  ONR  NR  049-317, 
April  1972. 

Spetzler*  C.XS?  The  Explicit  Consideration  of  Uncertainty  in  Capital 
Investment  Analysis,  Ph.D.  Dissertation,  Ill.  Inst,  of  Tech., 

Feb.  1968. 

Srinivasan,  T.  Development  of  an  Objective  Function  for  System 
Evaluation,  Masters  Thesis,  Dept,  of  Industrial  Engineering, 
Lehigh  University,  May  1972. 

i 

Stevens,  S.  S.  "Measurement,  Psychophysics,  and  Utility,"  Ch.  2, 
Measurement:  Definitions  and  Theories.  (Churchman,  C.  W.  and 
Ratoosh,  P.,  (ed.)).  New  York:  Wiley,  1959. 

Stimson,  D.  II.  "Utility  Measurement  in  Public  Health  Decision 

Making,"  Management  Science,  Vol.  16,  No.  2,  (Oct.  1969),  p.  317. 

Stokes,  P.  M.  A  Total  Systems  Approach  to  Management  Control. 

American  Management  Association,  1968. 

Stuff lebeam,  D.  L.  Educational  Evaluation  and  Decision-Making. 

Itasca,  Ill.:  F.  E.  Peacock,  1971. 

Swalm,  R.  0.  "Utility  Theory-Insights  into  Risk  Taking,"  Harvard 
Business  Review,  (Nov. -Dec.  1966),  pp.  123-36. 

Taylor,  A.;  Hillegass,  J.  R.;  and  Statland,  N.  Quantitative  Methods 
for  Information  Processing  Systems  Evaluation,  Report  No.  ESD- 
TDR-64-194,  (DDC  No.  AD  435  557),  Phila.,  Pa.:  Auerbach  for 
USAF,  Jan.  1964. 

Thorndike,  R.  L.  and  Hagen,  E.  Measurements  and  Evaluations  in 
Psychology  and  Education.  New  York:  Wiley,  1961. 


IV  -  9 


V  >,?>«» AW  \m  , 


0 

0 

0 

u 

y 

u 

LI 

U 

l! 


I! 

y 

IJ 

LI 

0 


0 

0 

0 


Turban,  E.  and  Metersky,  M.  L.  "Utility  Theory  Applied  to  Multi- 
voriate  System  Effectiveness  Evaluation,"  Management  Science, 
Vol.  17,  No.  12  (Aug.  1971),  pp.  B817-28. 

Westat  Research  Inc.  Procedural  Guide  for  the  Evaluation  of  Docu¬ 
ment  Retrieval  Systems.  (NSF-C491,  PB  182  711)  Washington, 
D.C.,  Dec.  31,  1968. 


Economic  Aspects 

Barish,  N.  N.  Economic  Analysis  for  Engineering  and  Managerial  De¬ 
cision-Making,  New  York:  McGraw-Hill,  1962. 

Bickner,  R.  E.  "Concepts  of  Economic  Cost,"  Ch.  3  of  Cost  Consider¬ 
ations  in  Systems  Analysis.  (Fisher,  G.  H.  (ed.)).  New  York: 
American  Elsevier,  1971. 

Blanchard,  B.  S.  "Cost  Effectiveness  Analysis  -  A  Case  Study  Approach," 
Annual  Technical  Conferee e.-- Transactions,  23rd  Conference.  Amer- 
lean  Society  for' Quality  Control,  pp.  229-38 . 


Boyd,  D.  F.  and  Krasnow,  H.  S.  "Economic  Evaluation  of  Management 
Information  Systems,"  IBM  Systems  Journal  (March  1963),  pp.  2-21. 

Chervany,  N.  L.  and  Dickson,  G.  W.  Economic  Evaluation  of  Management 
Information  Systems:  An  Analytical "Framework.  (MS IRC -A 69 -6)"  MIS 
Research  Center,  Univ.  of  Minn.,  1969. 

Fleisher,  G.  A.  and  Cremer,  R.  H.  "On  the  Application  of  Cardinal 
Utility,"  Engineering  Economist,  Vol.  16,  No.  2  (Winter  1971), 
pp.  117-130. 

Hoag,  M.  W.  "The  Relevance  of  Costs,"  Ch.  6,  Analysis  for  Military 
Decisions.  (Quade,  E.  S.  (ed.).  New  York:  North  Holland  and  Rand 
McNally,  1964. 

Jones,  P.  A.  "The  Computer:  A  Cost-Benefit  Analysis,"  Management 
Accounting,  Vol.  53,  No.  7  (July  1971),  pp.  23-5. 

McDonough,  A.  M.  "Values  and  Measurement,"  Ch.  9,  Information 

Economics  and  Management  Systems.  New  York:  McGraw-Hill,  1969. 

Nunamaker,  Jr.  J.  and  Whinster,  A.  Design  of  a  Corporate  Computer 
from  Problem  Statement  to  Cost  Allocation.  (DDC  No.  AD  724  161), 
Krannert  Graduate  School  of  Industrial  Administration,  April  1971. 

Schwab,  B.  "The  Economics  of  Sharing  Computers,"  Harvard  Business 
Review,  Vol.  46,  No.  5  (Sept. -Oct.  1968),  p.  61. 


47 

IV  - 10 


Seiler,  K.  Introduction  to  Systems  Cost-Effectiveness.  New  Yorks 
Wiley,  1969. 


Sharpe,  Wm*  F.  The  Economics  of  Computers.  New  York:  Columbia  Univ. 
Press,  1969. 

Taylor,  G.  A.  Managerial  and  Engineering  Economy;  Economic  Decision- 
Making.  Princeton,  N.J.:  Van  Nostrand,  1964. 

Theil,  H.  "The  Information  Concept,"  Ch.  1,  Economics  and  Information 
Theory.  New  York:  North  Holland  and  Rand  McNally,  1967. 

Zani,  Wm.  N.  "Real-time  Information  Systems:  A  Comparative  Economic 
Analysis,"  Management  Science,  (Feb.  1970),  pp.  350-5. 


Behavioral  Aspects 

- E -  * 

Caplan,  E.  H.  "Behavioral  Assumptions  of  Management  Accounting," 
Accounting  Review,  Vol.  16  (July  1966),  pp.  496-509. 

Carzo,  Jr.  R.  and  Yanouzas,  J.  N.  "Organizational  Design,"  Part  3, 
Formal  Organization  —  A  Systems  Approach,  Homewood,  Ill.: 

Irwin,  1967. 

Coombs,  C.  H.  and  Beardslee,  D.  "On  Decision-Making  Under  Uncertainty," 
Ch.  17,  Decision  Processes,  (Thrall,  R.  M.  et  al.  (ed.)),  New  York: 
Wiley,  1954. 

Gagne,  R.  M.  (ed.)  Psychological  Principles  in  System  Development. 

New  York:  Holt,  Rinehart  and  Winston,  1962. 

Glaser,  R.  and  Klaus,  D.  J.  "Proficiency  Measurement:  Assessing  Human 
Performance,"  Ch.  12,  Psychological  Principles  in  System  Develop¬ 
ment.  (Gagne,  R.  M.  (ed.)),  New  York:  Holt,  Rinehart  and  Winston, 
1962. 

Ijiri,  Yujij  Jaedicke,  R.  K.  and  Knight,  K.  E.  "The  Effects  of  Ac¬ 
counting  Alternatives  on  Management  Decisions,"  from  Research  in 
Accounting  Measurement.  Jaedicke,  R.  K.,  Ijiri,  Y.  and  Nielson, 

0.  (ei'T)),  Chicago:  American  Accounting  Association,  1966,  pp. 
186-99. 

Kast,  F.  E.  and  Rosenzweig,  J.  .  "Managerial  Information-Decision 
Systems,"  Ch.  12,  "Behavioral  Aspects  of  Decision-Making,"  Ch.  14, 
Organization  and  Management  —  A  Systems  Approach.  New  York: 
McGraw-Hill,  1970. 

Likert,  R.  "Time:  A  Key  Variable  in  Evaluating  Management  Systems," 

Ch.  5,  "Measurement,"  Ch.  8,  The  Human  Organization:  Its  Manage¬ 
ment  and  Value.  New  York:  McGraw-Hill,  1967. 


48 

iv  - 11 


Miller,  E.  C.  Objectives  and  Standards,  AMA  Research  Report  74, 

New  York:  American  Management  Association,  Jan.  1966. 

Miller,  R.  B.  "Task  Description  and  Analysis,"  Ch.  6,  Psychological 
Principles  in  System  Development.  (Gagne,  R.  M.  (ed.) ),  New  York 
Holt,  Rinehart  and  Winston,  1962,  pp.  187-228. 

Rappaport,  A.  Information  for  Decision-Making:  Quantitative  and  Be¬ 
havioral  Dimensions.  Englewood  Cliffs,  N.  J.:  Prentice-Hall, 
1970. 

Richter,  M.  Psychological  Assumptions  in  Political  Decision-Making. 
Paper  presented  at  79th  Annual  Convention  of  the  American  Psycho¬ 
logical  Association  Symposium.  Washington,  D.  C.  Sept.  4,  1971. 

Ridgway,  V.  F.  "Dysfunctional  Consequences  of  Performance  Measure¬ 
ments,"  Administrative  Science  Quarterly,  Vol.  I  (Sept.  1956), 
pp.  240-7. 

Rosove,  P.  E.  Developing  Computer-Based  Information  Systems .  New 
York:  Wiley,  1967.  *  >  *  *  * 

Seiler,  J.  E.  Systems  Analysis  in  Organizational  Behavior.  Homewood 
Ill.:  Irwin,  1967. 

Schroder,  H.  M.,  Driver,  M.  J.  and  Streuffort,  S.  Human  Information 
Processing.  Mew  York:  Holt,  Rinehart  and  Winston  1967. 

Thorndike,  R.  L.  and  Hagen,  E.  Measurements  and  Evaluation  in 
Psychology  and  Education.  New  York:  Wiley,  1961. 


49 

IV  -  12 


V.  Areas  of  Research 


This  paper  has  stressed  both  the  difficulties  in  measuring  in¬ 
formation  system  effectiveness  and  the  fact  that  it  is  not  an  isolated 
problem.  A  range  of  characteristics  and  purposes  can  be  attributed 
to  an  "information  system."  A  proposed  decision  model  sketches  the 
dynamic,  intermittent  interactions  between  the  system  and  the  decision 
process  which  it  serves.  Dependence  upon  organization  structure  and 
style  is  indicated.  Man-machine  system  concepts  apply  in  a  synergism 
of  computer  programs  and  manual  procedures.  Individual  differences 
among  users  and  shifts  in  operational  priorities  create  a  fluctuating 
demand  for  content,  form  and  amount  of  information.  In  all,  the  con¬ 
cept  of  information  quality  requires  further  exploration  and  definition 
in  terms  of  the  operational  environment. 

Progress  toward  a  more  comprehensive  system  theory  would  provide 

%  • 

greater  insight  about  the  role  of  information  in  purposive  systems. 
Proven  methods  for  analysis  and  design  of  systems  and  programs  are 
far  from  uniformly  accepted  and  applied.  The  major  phases  of  informa¬ 
tion  processing  themselves  suggest  many  practical  topics  which  will 
benefit  from  continuing  applied  research: 

Phase  Example  topics 


Data  gathering 
Data  recording 
Processing 


Storage 

Dissemination 


Errors,  instrumentation 
Conversion  process,  media 
Hardware/software  selection,  operating 
system,  program  structure,  data  flow, 
performance  criteria,  cost  allocation, 
logic  development 
File  structure,  data  management 
Media,  method,  format 


The  principal  concern  addressed  in  this  manual,  however,  has  been  the 
use  and  impact  of  information  system  output.  The  areas  outlined  in 
the  following  sections  are  suggested  for  separate  study,  for  collating 
into  a  procedure,  or  for  transfer  of  application  from  other  problems 
to  those  related  to  information  systems. 
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Study  Approaches 

Methods  of  measuring  effectiveness  are  included  for  investigation 
because  of  their  important  use  in  the  conduct  of  research,  but  also 
because  they  should  be  the  object  of  further  development.  More  should 
be  known  about  the  most  appropriate  method  for  a  given  set  of  circum¬ 
stances.  Analytical  models  may  prove  adequate  and  appropriate  for 
some  aspects  of  systems.  It  is  more  likely  that  simulation  models  will 
be  more  successful  because  of  the  dynamic,  complex  nature  of  the  per¬ 
formance  encountered.  Field  study  is  quite  common  and  takes  several 
forms : 

-  Evaluation  studies  (Arinc;  SDC;  Smith  and  Wolf;  Wes tat) 

-  Performance  measurement  (Karsen,  Karush,  Knight,  Chapter  II) 

-  Observation  and  analysis  of  activity  (Heiland  and  Richardson, 
Reuter,  Richardson) 

-  Judgment  surveys.  Adaptations  of  the  Delphi  technique  will 
prove  useful  to  gain  consensus  about  ill  defined  objectives 
and  about  opinions  concerning  effectiveness  from  differently 
biased  observers.  (Smith  and  Wolf) 

Experimental  laboratory  approaches  are  possible  but,  because  of  the 
behavioral  involvements,  they  are  usually  linked  with  gaming  approaches 
(Barkin;  Cohen  and  Van  Horn;  Davis  and  Behan;  Lopez;  Robinson  and 
Stidsen).  Particular  attention  should  be  given  to  developing  in- 
basket  techniques  both  as  a  tool  for  analysis  of  operational  situa¬ 
tions  and  for  experimentation  with  and  observation  of  performance 
under  controlled  conditions.  Despite  the  easy  reference  to  scenarios 
for  such  studies,  there  is  little  evidence  about  success  in  their  use. 
Guidelines  for  writing  and  validating  them  would  be  valuable  contri¬ 
butions  . 

Human  Behavior 

The  human  is  critical  to  information  system  effectiveness  in  his 
several  roles  as  designer,  implementer,  component,  and  user.  How 
does  one  classify  the  nature  and  extent  of  a  change  in  behavior  which 
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results  from  receiving  information?  Should  output  be  factual  as  in 
an  administrative  traffic  system  or  should  it  be  influential  as  in  a 
political  message?  The  intermittent  receipt  of  information  suggests 
issues  related  to  learning,  heuristics  and  group  dynamics.  Engineer¬ 
ing  psychology  and  the  ability  of  the  human  as  an  information  proces¬ 
sor  relate  to  the  output  a  system  should  or  can  deliver. 

The  area  of  decision  making  deserves  particular  research  atten¬ 
tion,  primarily  in  such  areas  as  criterion  formulation,  utility  and 
risk  attitudes,  reinforcement,  preference  ordering,  and  choice  selec¬ 
tion.  Of  growing  importance  will  be  the  mutual  considerations  of 
organization  and  information  system  design  (Carzo  and  Yanouzas). 

Cost/Benefits 

The  central  issue  of  the  effectiveness  problem  is  the  comparison 
of  costs  and  benefits.  As  we  have  indicated  throughout  the  paper,  the 
benefits  are  commonly  intangible  and  difficult  to  agree  upon  or  mea¬ 
sure.  Although  costs  have  been  given  attention  from  the  accounting 
viewpoint  for  many  years,  allocation  procedures  are  not  well-defined 
in  complex  situations.  Broader  use  of  systems  engineering  approaches, 
stressing  dominance  of  overall  organization  goals,  should  clarify 
many  points  in  this  area.  Both  education  of  managers  and  more  flexible 
tools  to  assist  application  are  necessary.  Cost  effectiveness  and 
PPBS  must  become  managerial  attitudes  rather  than  narrowly  applied  pro¬ 
cedures.  Positive  and  negative  effects  of  information  systems  must 
be  translated  into  a  net  or  aggregate  of  benefits.  Costs,  including 
sociological  scars  and  political  failures,  must  be  drawn  into  com¬ 
parable  terms  for  analysis.  The  size  and  nature  of  systems  discussed 
will  mean  long  periods  of  development  and  maintenance.  Greater  con¬ 
sideration  needs  to  be  given  to  the  worth  of  intermediate  goals, 
amortized  values,  and  extent  of  commitment,  both  management  and  re* 
sources,  implied  by  embarking  on  the  extended  euphoric  trip  to  better 
information. 

Models 

Stochastic  models  may,  in  some  cases,  unlock  part  of  the  artistic 
areas  of  information  system  design  and  allow  progress  toward  more 
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scientific  approaches.  Bayesian  models  and  subjective  aspects  of 
probability  are  of  particular  concern  to  the  decision  process  being 
served. 

The  problems  of  describing  and  analyzing  flows  of  data  in  an 
organization  remain  despite  several  furtive  contributions.  Matrix 
approaches  have  proved  to  be  computationally  feasible  for  a  relative¬ 
ly  small  number  of  data  elements  and  functional  activities.  (Homer; 
Taylor,  Schmidt  and  Ghare;  Stults;  Wilson  and  Smith).  The  effects 
of  data  omissions  and  errors  on  time  and  quality  need  greater  under¬ 
standing.  Network  approaches  have  the  potential  to  deal  with  feedback, 
time  relationships,  and  reliability  of  events.  Both  GERT  and  Indus¬ 
trial  Dynamics  should  be  explored  to  assess  their  full  capability  in 
measuring  effectiveness  (Roberts,  Forrester,  Pritsker  and  Whitehouse). 

'  Optimization  and  mathematical  programming  approaches  are  applied 
to  many  systems  with  more  tangible  and  finite  output.  Both  limita¬ 
tions  and  possible  application  of  the  various  techniques  should  be 
explored.  Also,  algebraic  approaches  to  solution  of  information  sys¬ 
tem  design  problems  should  be  considered  further  (Li).  One  particular 
concept,  fuzzy  sets,  has  some  appeal  when  dealing  with  partial  or  ag¬ 
gregate  specifications,  changing  criteria  and  grouping  of  attributes, 
and  ranges  of  values  measured  (Zadeh,  Hormann). 

Theory 

Uncertainty  and  risk  are  areas  which  need  to  be  understood 
thoroughly  by  an  information  system  designer.  Decision  theory  es¬ 
tablishes  some  norms  that  are  useful  in  concept.  But  it  appears  tc 
have  limited  practical  value  in  relation  to  the  many  uncontrolled 
variables  encountered  in  most  operational  situations.  Decision  anal¬ 
ysis  bridges  the  gap  and  deals  more  directly  with  the  subjective 
probabilities  related  to  uncertain  situations.  Utility  and  value 
theories,  on  the  other  hand,  are  more  descriptive  of  the  acquisition 
and  utilization  of  information  of  various  kinds  by  a  decision-maker. 
These  theories  suggest  a  static  individual  characteristic  which  is 
not  consistent  with  observed  performance.  The  need  to  recognize  and 
accommodate  different  preference  sets,  based  on  variations  in 
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experience  and  environmental  influence  is  apparent.  Relationships 
between  individual  and  group  utility  also  need  exploration. 

Particular  attention  needs  to  be  given  to  communication  theory  (a 
label  I  prefer  to  information  theory  for  Shannon’s  work).  Despite  its 
limitation  relative  to  influencing  action,  the  entropy  and  amount  of 
information  concepts  have  not  been  exploited  to  the  advantage  of  better 
system  design  or  performance  (Marschak).  The  multistage  aspects  of  data 
processing,  information  flow  and  decision  making  require  much  greater 
attention  to  control  theory  and  Markhov  processes  (Hodge  and  Hodgson). 
The  latter  are  important,  not  just  because  information  should  aid 
control  over  activity,  but  primarily  because  the  system  is  a  process 
which  itself  must  be  subject  to  sophisticated  control. 

Although  game  theory  has  had  little  practical  application,  it  or 
an  adaptation  may  become  an  aid  to  the  operators  of  a  system  for  de¬ 
scribing  or  limiting  its  capabilities.  Efficiency  of  perf ormance/ and 
balance  of  system  configuration  depend  upon  understanding  and  apply¬ 
ing  queueing  theory.  Users,  transactions  and  problems  all  develop 
their  waiting  line  and  service  profiles  which  vary  widely.  Pattern 
recognition  and  related  cognitive  and  perception  theories  have  poten¬ 
tial  for  contribution  in  relation  to  media  or  format  selection  and 
to  development  of  order  and  simplicity  in  human-system  interactions. 

A  final  area  for  additional  research  is  related  to  sensitivity 
analysis.  This  is  crucial  if  cause-effect  relationships  are  to  be 
identified.  Reactions  to  faulty  input,  to  normal  fluctuations  in  in¬ 
put  or  processing,  and  to  environmental  influences  must  be  more  easily 
gauged.  This  requires  investigation  and  improvement  of  measuring  and 
scaling  techniques  for  both  quantitative  and  qualitative  variables. 

It  is  imperative  to  have  better  ways  to  define  the  independent  and 
dependent  variables,  to  test  the  relationships,  and  to  provide  repre¬ 
sentative  data  for  study  purposes.  Knowing  the  causal  and  magnitude 
relationships  of  effects  will  allow  tuning  of  system  performance  by 
controlling  variables  selected  during  design. 
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